Digital television rendering verification

ABSTRACT

Techniques are described for expanding and/or improving the Advanced Television Systems Committee (ATSC) 3.0 television protocol in robustly delivering the next generation broadcast television services. Test code is embedded in ATSC 3.0 content to determine whether a receiver can render the content. The results are reported back to the broadcaster, which can alter content or perform other modifications based on aggregated rendering statistics.

FIELD

This application relates to technical advances necessarily rooted incomputer technology and directed to digital television, and moreparticularly to Advanced Television Systems Committee (ATSC) 3.0.

BACKGROUND The Advanced Television Systems Committee (ATSC) 3.0 suite ofstandards is a set of over a dozen industry technical standards asindicated in A/300 for delivering the next generation of broadcasttelevision. ATSC 3.0 supports delivery of a wide range of televisionservices including televised video, interactive services, non-real timedelivery of data, and tailored advertising to a large number ofreceiving devices, from ultra-high definition televisions to wirelesstelephones. ATSC 3.0 also orchestrates coordination between broadcastcontent (referred to as “over the air”) and related broadband deliveredcontent and services (referred to as “over the top”). ATSC 3.0 isdesigned to be flexible so that as technology evolves, advances can bereadily incorporated without requiring a complete overhaul of anyrelated technical standard. Present principles are directed to suchadvances as divulged below.

SUMMARY

As understood herein, it is desirable to ensure that a receiver canproperly render ATSC 3.0 content in accordance with the appropriate ATSC3.0 standard, e.g., in accordance with A/344 (referencing ConsumeTechnology Association (CTA) 5000). This is to eliminate or limit roguedevices that otherwise might receive ATSC 3.0 content and redistributeit in ways that would deprive intended beneficiaries of revenue.

Accordingly, a digital television system includes at least one receiverwith at least one processor programmed with instructions to configurethe processor to receive from broadcaster equipment a test application,and to execute the test application. The instructions are executable toimplement an application programming interface (API) associated withhypertext markup language (HTML) Entry pages Location Description (HELD)signaling for logging commands to and responses from a device undertest. The instructions also are executable to, based at least in part onthe HELD signaling, load at least one entry page from at least one webserver onto at least one browser of the receiver, and using the testapplication, execute API calls to exercise compliance with a standard.

The test application is a specialized broadcaster application thatself-generates commands as part of a rendering test. The testapplication self-generates commands to send, such as but not limited tochannel change commands, and monitors for correct feedback, e.g.,correct channel change. Essentially the test application in the receiverdevice under test executes a test script.

In some example implementations, the instructions can be executable toexecute the test application to exercise at least one connection to atleast one WebSocket Server, at least one hypertext transfer protocol(HTTP) interface, and reception of the HELD signaling.

In non-limiting embodiments, the instructions may be executable toexecute the test application to test support of a user agent of thereceiver for at least one web browser using reception of the HELDsignaling at least on part by sending a signal back through a hypertexttransfer protocol (HTTP) interface to a test computer.

If desired, the instructions can be executable to execute the testapplication to test for proper HELD signaling at least in part bysending a ‘hello’ signal back to the broadcaster equipment. In exampleembodiments the instructions may be executable to execute the testapplication to test for proper HELD signaling at least in part bymaintaining in storage of the receiver a ‘hello’ signal. Moreover, insome implementations the instructions are executable to execute the testapplication to identify proper execution of APIs of at least oneWebSocket of the receiver connected to at least one web socket server.

In an example, at least one and in some non-limiting embodiments one andonly one test application is executed by the processor and theinstructions are executable to use the one and only one test applicationto execute a first API command associated with a first broadcast channeland execute a second API command associated with a second broadcastchannel to identify execution of a request to change tuning from thefirst broadcast channel to the second broadcast channel.

In another aspect, a digital television system includes at least onereceiver configured to receive digital television from at least onetransmitter assembly. The receiver includes at least one processorprogrammed with instructions to execute at least a first broadcasterapplication to access test code in at least a first service, identify,based at least in part on the test code, whether the receiver renderscontent, and transmit a signal to the transmitter assembly indicatingwhether the receiver renders the content.

The test code may include, for example, cascading style sheet (CSS)code. The test code may include a command to change service from a firstservice to a second service, and the instructions can be executable touse a first application programming interface (API) command in renderingthe first service and a second API command in rendering the secondservice.

In another aspect, in a digital television system, a method includesexecuting at least a first broadcaster application to access test codein at least a first service provided to a receiver. The method alsoincludes identifying, based at least in part on the test code, whetherthe receiver renders content, and transmitting a signal to a transmitterassembly indicating whether the receiver renders the content.

The details of the present application, both as to its structure andoperation, can best be understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an Advanced Television Systems Committee(ATSC) 3.0 system;

FIG. 2 is a block diagram showing components of the devices shown inFIG. 1;

FIG. 3 is a block diagram of a simplified test architecture;

FIG. 4 is a block diagram illustrating verification test techniques; and

FIGS. 5 and 6 are flow charts of example logic consistent with presentprinciples.

DETAILED DESCRIPTION

This disclosure relates to technical advances in digital television suchas in Advanced Television Systems Committee (ATSC) 3.0 television. Anexample system herein may include ATSC 3.0 source components and clientcomponents, connected via broadcast and/or over a network such that datamay be exchanged between the client and ATSC 3.0 source components. Theclient components may include one or more computing devices includingportable televisions (e.g. smart TVs, Internet-enabled TVs), portablecomputers such as laptops and tablet computers, and other mobile devicesincluding smart phones and additional examples discussed below. Theseclient devices may operate with a variety of operating environments. Forexample, some of the client computers may employ, as examples, operatingsystems from Microsoft, or a Unix operating system, or operating systemsproduced by Apple Computer or Google, such as Android®. These operatingenvironments may be used to execute one or more browsing programs, suchas a browser made by Microsoft or Google or Mozilla or other browserprogram that can access websites hosted by the Internet serversdiscussed below.

ATSC 3.0 source components may include broadcast transmission componentsand servers and/or gateways that may include one or more processorsexecuting instructions that configure the source components to broadcastdata and/or to transmit data over a network such as the Internet.

A client component and/or a local ATSC 3.0 source component may beinstantiated by a game console such as a Sony PlayStation®, a personalcomputer, etc.

Information may be exchanged over a network between the clients andservers. To this end and for security, servers and/or clients caninclude firewalls, load balancers, temporary storages, and proxies, andother network infrastructure for reliability and security.

As used herein, instructions refer to computer-implemented steps forprocessing information in the system. Instructions can be implemented insoftware, firmware or hardware and include any type of programmed stepundertaken by components of the system.

A processor may be any conventional general-purpose single- ormulti-chip processor that can execute logic by means of various linessuch as address lines, data lines, and control lines and registers andshift registers.

Software modules described by way of the flow charts and user interfacesherein can include various sub-routines, procedures, etc. Withoutlimiting the disclosure, logic stated to be executed by a particularmodule can be redistributed to other software modules and/or combinedtogether in a single module and/or made available in a shareablelibrary. While flow chart format may be used, it is to be understoodthat software may be implemented as a state machine or other logicalmethod.

Present principles described herein can be implemented as hardware,software, firmware, or combinations thereof; hence, illustrativecomponents, blocks, modules, circuits, and steps are set forth in termsof their functionality.

Further to what has been alluded to above, logical blocks, modules, andcircuits can be implemented or performed with a general-purposeprocessor, a digital signal processor (DSP), a field programmable gatearray (FPGA) or other programmable logic device such as an applicationspecific integrated circuit (ASIC), discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A processor can be implementedby a controller or state machine or a combination of computing devices.

The functions and methods described below, when implemented in software,can be written in an appropriate language such as but not limited tohypertext markup language (HTML)-5, Java®/Javascript, C# or C++, and canbe stored on or transmitted through a computer-readable storage mediumsuch as a random access memory (RAM), read-only memory (ROM),electrically erasable programmable read-only memory (EEPROM), compactdisk read-only memory (CD-ROM) or other optical disk storage such asdigital versatile disc (DVD), magnetic disk storage or other magneticstorage devices including removable thumb drives, etc. A connection mayestablish a computer-readable medium. Such connections can include, asexamples, hard-wired cables including fiber optics and coaxial wires anddigital subscriber line (DSL) and twisted pair wires.

Components included in one embodiment can be used in other embodimentsin any appropriate combination. For example, any of the variouscomponents described herein and/or depicted in the Figures may becombined, interchanged or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system havingat least one of A, B, or C” and “a system having at least one of A, B,C”) includes systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.

Turning to FIG. 1, an example of an ATSC 3.0 source component is labeled“broadcaster equipment” 10 and may include over-the-air (OTA) equipment12 for wirelessly broadcasting, typically via orthogonal frequencydivision multiplexing (OFDM) in a one-to-many relationship, televisiondata to plural receivers 14 such as ATSC 3.0 televisions. One or morereceivers 14 may communicate with one or more companion devices 16 suchas remote controls, tablet computers, mobile telephones, and the likeover a short range, typically wireless link 18 that may be implementedby Bluetooth®, low energy Bluetooth, other near field communication(NFC) protocol, infrared (IR), etc.

Also, one or more of the receivers 14 may communicate, via a wiredand/or wireless network link 20 such as the Internet, with over-the-top(OTT) equipment 22 of the broadcaster equipment 10 typically in aone-to-one relationship. The OTA equipment 12 may be co-located with theOTT equipment 22 or the two sides 12, 22 of the broadcaster equipment 10may be remote from each other and may communicate with each otherthrough appropriate means. In any case, a receiver 14 may receive ATSC3.0 television signals OTA over a tuned-to ATSC 3.0 television channeland may also receive related content, including television, OTT(broadband). Note that computerized devices described in all of thefigures herein may include some or all of the components set forth forvarious devices in FIGS. 1 and 2.

Referring now to FIG. 2, details of examples of components shown in FIG.1 may be seen. FIG. 2 illustrates an example protocol stack that may beimplemented by a combination of hardware and software. Using the ATSC3.0 protocol stack shown in FIG. 2 and modified as appropriate for thebroadcaster side, broadcasters can send hybrid service delivery in whichone or more program elements are delivered via a computer network(referred to herein as “broadband” and “over-the-top” (OTT)) as well asvia a wireless broadcast (referred to herein as “broadcast” and“over-the-air” (OTA)). FIG. 2 also illustrates an example stack withhardware that may be embodied by a receiver.

Disclosing FIG. 2 in terms of broadcaster equipment 10, one or moreprocessors 200 accessing one or more computer storage media 202 such asany memories or storages described herein may be implemented to provideone or more software applications in a top-level application layer 204.The application layer 204 can include one or more software applicationswritten in, e.g., HTML5/Javascript running in a runtime environment.Without limitation, the applications in the application stack 204 mayinclude linear TV applications, interactive service applications,companion screen applications, personalization applications, emergencyalert applications, and usage reporting applications. The applicationstypically are embodied in software that represents the elements that theviewer experiences, including video coding, audio coding and therun-time environment. As an example, an application may be provided thatenables a user to control dialog, use alternate audio tracks, controlaudio parameters such as normalization and dynamic range, and so on.

Below the application layer 204 is a presentation layer 206. Thepresentation layer 206 includes, on the broadcast (OTA) side, broadcastaudio-video playback functions referred to as Media Processing Units(MPU) 208 that, when implemented in a receiver, decode and playback, onone or more displays and speakers, wirelessly broadcast audio videocontent. The MPU 208 is configured to present International Organizationfor Standardization (ISO) base media file format (BMFF) datarepresentations 210 and video in high efficiency video coding (HEVC)with audio in, e.g., Dolby audio compression (AC)-4 format. ISO BMFF isa general file structure for time-based media files broken into“segments” and presentation metadata. Each of the files is essentially acollection of nested objects each with a type and a length. Tofacilitate decryption, the MPU 208 may access a broadcast side encryptedmedia extension (EME)/common encryption (CENC) module 212.

FIG. 2 further illustrates that on the broadcast side the presentationlayer 206 may include signaling modules, including either motionpictures expert group (MPEG) media transport protocol (MMTP) signalingmodule 214 or real-time object delivery over unidirectional transport(ROUTE) signaling module 216 for delivering non-real time (NRT) content218 that is accessible to the application layer 204. NRT content mayinclude but is not limited to broadcaster applications or storedreplacement advertisements.

On the broadband (OTT or computer network) side, when implemented by areceiver the presentation layer 206 can include one or more dynamicadaptive streaming over hypertext transfer protocol (HTTP) (DASH)player/decoders 220 for decoding and playing audio-video content fromthe Internet. To this end the DASH player 220 may access a broadbandside EME/CENC module 222. The DASH content may be provided as DASHsegments 224 in ISO/BMFF format.

As was the case for the broadcast side, the broadband side of thepresentation layer 206 may include NRT content in files 226 and may alsoinclude signaling objects 228 for providing play back signaling.

Below the presentation layer 206 in the protocol stack is a sessionlayer 230. The session layer 230 includes, on the broadcast side, eitherMMTP protocol 232 or ROUTE protocol 234.

On the broadband side the session layer 230 includes HTTP protocol 236which may be implemented as HTTP-secure (HTTP(S). The broadcast side ofthe session layer 230 also may employ a HTTP proxy module 238 and aservice list table (SLT) 240. The SLT 240 includes a table of signalinginformation which is used to build a basic service listing and providediscovery of the broadcast content. Media presentation descriptions(MPD) are included in the “ROUTE Signaling” tables delivered over userdatagram protocol (UDP) by the ROUTE transport protocol.

A transport layer 242 is below the session layer 230 in the protocolstack for establishing low-latency and loss-tolerating connections. Onthe broadcast side the transport layer 242 uses (UDP) 244 and on thebroadband side transmission control protocol (TCP) 246.

The example non-limiting protocol stack shown in FIG. 2 also includes anetwork layer 248 below the transport layer 242. The network layer 248uses Internet protocol (IP) on both sides for IP packet communication,with multicast delivery being typical on the broadcast side and unicastbeing typical on the broadband side.

Below the network layer 248 is the physical layer 250 which includesbroadcast transmission/receive equipment 252 and computer networkinterface(s) 254 for communicating on the respective physical mediaassociated with the two sides. The physical layer 250 converts InternetProtocol (IP) packets to be suitable to be transport over the relevantmedium and may add forward error correction functionality to enableerror correction at the receiver as well as contain modulation anddemodulation modules to incorporate modulation and demodulationfunctionalities. This converts bits into symbols for long distancetransmission as well as to increase bandwidth efficiency. On the OTAside the physical layer 250 typically includes a wireless broadcasttransmitter to broadcast data wirelessly using orthogonal frequencydivision multiplexing (OFDM) while on the OTT side the physical layer250 includes computer transmission components to send data over theInternet.

A DASH Industry Forum (DASH-IF) profile sent through the variousprotocols (HTTP/TCP/IP) in the protocol stack may be used on thebroadband side. Media files in the DASH-IF profile based on the ISO BMFFmay be used as the delivery, media encapsulation and synchronizationformat for both broadcast and broadband delivery.

Each receiver 14 typically includes a protocol stack that iscomplementary to that of the broadcaster equipment.

A receiver 14 in FIG. 1 may include, as shown in FIG. 2, anInternet-enabled TV with an ATSC 3.0 TV tuner (equivalently, set top boxcontrolling a TV) 256. The receiver 14 may be an Android®-based system.The receiver 14 alternatively may be implemented by a computerizedInternet enabled (“smart”) telephone, a tablet computer, a notebookcomputer, a wearable computerized device, and so on. Regardless, it isto be understood that the receiver 14 and/or other computers describedherein is configured to undertake present principles (e.g. communicatewith other devices to undertake present principles, execute the logicdescribed herein, and perform any other functions and/or operationsdescribed herein).

Accordingly, to undertake such principles the receiver 14 can beestablished by some or all of the components shown in FIG. 1. Forexample, the receiver 14 can include one or more displays 258 that maybe implemented by a high definition or ultra-high definition “4K” orhigher flat screen and that may or may not be touch-enabled forreceiving user input signals via touches on the display. The receiver 14may also include one or more speakers 260 for outputting audio inaccordance with present principles, and at least one additional inputdevice 262 such as, e.g., an audio receiver/microphone for, e.g.,entering audible commands to the receiver 14 to control the receiver 14.The example receiver 14 may further include one or more networkinterfaces 264 for communication over at least one network such as theInternet, a WAN, a LAN, a PAN etc. under control of one or moreprocessors 266. Thus, the interface 264 may be, without limitation, aWi-Fi transceiver, which is an example of a wireless computer networkinterface, such as but not limited to a mesh network transceiver. Theinterface 264 may be, without limitation, a Bluetooth® transceiver,Zigbee® transceiver, Infrared Data Association (IrDA) transceiver,Wireless USB transceiver, wired USB, wired LAN, Powerline or Multimediaover Coax Alliance (MoCA). It is to be understood that the processor 266controls the receiver 14 to undertake present principles, including theother elements of the receiver 14 described herein such as, forinstance, controlling the display 258 to present images thereon andreceiving input therefrom. Furthermore, note the network interface 264may be, e.g., a wired or wireless modem or router, or other appropriateinterface such as, e.g., a wireless telephony transceiver, or Wi-Fitransceiver as mentioned above, etc.

In addition to the foregoing, the receiver 14 may also include one ormore input ports 268 such as a high definition multimedia interface(HDMI) port or a USB port to physically connect (using a wiredconnection) to another CE device and/or a headphone port to connectheadphones to the receiver 14 for presentation of audio from thereceiver 14 to a user through the headphones. For example, the inputport 268 may be connected via wire or wirelessly to a cable or satellitesource of audio video content. Thus, the source may be a separate orintegrated set top box, or a satellite receiver. Or, the source may be agame console or disk player.

The receiver 14 may further include one or more computer memories 270such as disk-based or solid-state storage that are not transitorysignals, in some cases embodied in the chassis of the receiver asstandalone devices or as a personal video recording device (PVR) orvideo disk player either internal or external to the chassis of thereceiver for playing back audio video (AV) programs or as removablememory media. Also, in some embodiments, the receiver 14 can include aposition or location receiver 272 such as but not limited to a cellphonereceiver, global positioning satellite (GPS) receiver, and/or altimeterthat is configured to e.g. receive geographic position information fromat least one satellite or cellphone tower and provide the information tothe processor 266 and/or determine an altitude at which the receiver 14is disposed in conjunction with the processor 266. However, it is to beunderstood that that another suitable position receiver other than acellphone receiver, GPS receiver and/or altimeter may be used inaccordance with present principles to determine the location of thereceiver 14 in e.g. all three dimensions.

Continuing the description of the receiver 14, in some embodiments thereceiver 14 may include one or more cameras 274 that may include one ormore of a thermal imaging camera, a digital camera such as a webcam,and/or a camera integrated into the receiver 14 and controllable by theprocessor 266 to gather pictures/images and/or video in accordance withpresent principles. Also included on the receiver 14 may be a Bluetooth®transceiver 276 or other Near Field Communication (NFC) element forcommunication with other devices using Bluetooth® and/or NFC technology,respectively. An example NFC element can be a radio frequencyidentification (RFID) element.

Further still, the receiver 14 may include one or more auxiliary sensors278 (such as a motion sensor such as an accelerometer, gyroscope,cyclometer, or a magnetic sensor and combinations thereof), an infrared(IR) sensor for receiving IR commands from a remote control, an opticalsensor, a speed and/or cadence sensor, a gesture sensor (for sensinggesture commands) and so on providing input to the processor 266. An IRsensor 280 may be provided to receive commands from a wireless remotecontrol. A battery (not shown) may be provided for powering the receiver14.

The companion device 16 may incorporate some or all of the elementsshown in relation to the receiver 14 described above.

The methods described herein may be implemented as software instructionsexecuted by a processor, suitably configured application specificintegrated circuits (ASIC) or field programmable gate array (FPGA)modules, or any other convenient manner as would be appreciated by thoseskilled in those art. Where employed, the software instructions may beembodied in a non-transitory device such as a CD ROM or Flash drive. Thesoftware code instructions may alternatively be embodied in a transitoryarrangement such as a radio or optical signal, or via a download overthe Internet.

Now referring to FIG. 3, a receiver 300 is shown. For test purposes, thereceiver 300 can be regarded as a device under test (DUT). The receiver300 includes a typically software-implemented user agent 302 that inturn includes a broadcaster application (BA) 304, which may beassociated with a single channel or a single subset of channels from thesame or related broadcasters. The BA 304 typically supports such ATSC3.0 functions as replacement content insertion into a broadcast receivedvia an ATSC 3.0 protocol stack 306. Consistent with principles describedabove, the replacement content may be received from the Internet 308 viaan HTTP interface 310, which may also receive information from a webserver 312 associated with the broadcaster for example having access toa cache 314 of content files in some cases received from the protocolstack 306. The BA 304 may communicate with the HTTP interface 310 usingJavascript application programming interfaces (API).

The BA 304 may further communicate with a web socket server 316 toaccess device resources 318, which may include a resident media player320 and a graphics engine 322. Communication between the BA 304 and websocket server 316 may be via Javascript object notation (JSON) remoteprocedure call (RPC) messages.

The example user agent 302 further may include a media source encryptionand/or encrypted media extension (MSE/EME) module 324.

With the architecture of FIG. 3 in mind, FIG. 4 illustrates a testenvironment for the DUT 300. Testing hybrid broadcast/broadbandconnectivity can be facilitated using a generic test BA 400. The test BA400 is essentially a test application that is configured with BAfunctionality but that is not provided by a broadcaster but rather by adevice manufacturer to run a test script in which the test application400 self-generates commands to components of the DUT and monitors forcorrect feedback.

The test application 400 is loaded onto the DUT 300 and essentiallyself-certifies responses from the ATSC 3.0 middleware 402 in the stack306 and data retrievals from the web server 312/web socket server 316.The DUT 300 can execute a browser of choice (such as those providedunder trademarks/trade names such as Firefox, Safari, Chrome, VEWD,etc.) that can execute HTML5 code. Furthermore, the DUT 300 executes aWeb Socket for Application Programming Interface (API) call handling toprovide information about the device itself, such as amount of memory isavailable, what video/audio codecs are available, etc.

The test application 400 can be loaded onto the DUT 300 via a PCAP file,designated by reference numeral 404 in FIG. 4, containing HTML Entrypages Location Description (HELD) signaling as described in ATSC 3.0standard A/331. More specifically, the test application 400 may be sentthrough the PCAP file 404 from the web server 312 to the DUT 300wirelessly, i.e., via ATSC 3.0 broadcast that is received by the ATSC3.0 physical layer 406 in the stack 306 of the DUT 300 and provided tothe middleware 402.

The HELD signaling triggers the DUT 300 to load an entry page(index.html) from, e.g., a web server onto its browser. The testapplication 400 then executes all API calls to exercise compliance witha given standard (e.g., A/344 or CTA5000).

The test application 400 exercises the connections to WebSocket Server316, the HTTP interface 310, and reception of the HELD signaling. TheWeb Socket interface contains the APIs for device resource reporting.The HTTP interface 310 can be tested for status report of so-called “404Not found” assets or other internet connection errors. Proper User Agent302 support of the web browser upon reception of the HELD signaling istested by sending a signal back through the HTTP interface 310 to a testcomputer that may be implemented in or by the web server 312.

The test environment shown in FIG. 4 includes a Web server such as theweb server 312 and a list of PCAP files to execute. In an examplenon-limiting implementation, the PCAP files can be loaded into a DekTectransmitter (such as an ATSC3Expert) and each one of the files can havea length that spans the duration of a test. In one example, a first testcan be for proper HELD signaling as indicated by a ‘hello’ signal sentback to the test environment via the HTTP interface 310 or a maintainedin a text log in the BA 400. These logs can be recorded on the DUT 300for later download or sent back to the test environment via the HTTPinterface 310.

After setup verification, the HTTP interface 310 can be tested by‘phoning home’ back to the test environment with a valid connectionstream, in some cases using the media access control (MAC) of the DUT300 or some other unique identifier for test logging.

Further tests can be conducted for proper execution of the APIs of theWeb Socket connected to the websocket server 316 and for properexecution of the browser of the DUT 300. In an example, APIs to test maybe those listed in the ATSC 3.0 standard A/344 or CTA5000 (bothincorporated herein by reference) or other tests that are deemednecessary to comply with NEXTGEN TV compliance.

For interactivity tests including accessing content from a testenvironment and timing aspects of tests (reply within a certain amountof time), loading of assets and responses from the DUT 300 can berecorded in the test application 400.

Having the test application 400 determine pass/fail for hybridinteractivity allows the testing to be cost effective and accurate.There is only one application 400 that need be loaded on the DUT 300 andthe list of tests remain the same. Updates to tests lists involve anupdate to the single test application. The test application 400 canhandle all interfaces and timing aspects of test criteria.

If an RF channel change is part of the tests, then a new PCAP file forthe new RF channel should contain the same HELD signaling for the sametest application with the same AppContextID so that the DUT can continuetesting. This allows interactive services like channel guides to betested as well.

FIGS. 5 and 6 provide further illustration. Commencing at block 500,test code such as any described herein is embedded into signals providedto the DUT 300. For example, the test code may include cascading stylesheet (CSS) code to be rendered by the DUT 300. The test code mayinclude a command to change service and the DUT may attempt to executethe command.

Moving to decision diamond 502, using, e.g., the PCAP file the DUT 300determines whether it has correctly rendered content in thesignals/correctly executed applications in the signals. The results arestored in the DUT for later proof of certification and/or provided tothe test broadcaster at block 504. If all tests pass the DUT 300 may bephysically labeled as a “NextGen TV” receiver.

If desired, the test broadcaster may aggregate test results from pluralDUTs at block 600 in FIG. 6. Moving to block 602, depending on theaggregated results the broadcasters may alter content format or not tocertain devices and/or device types. For example, if a DUT fails toprovide test results on demand as part of a certification process, thatdevice may be locked out of receiving ATSC 3.0 content by, e.g.,revoking security keys/certificates of the device.

Note that the above techniques may be used by the manufacturer prior tovending a receiver to ensure compliance, and also by an end user aftervending for debugging a receiver that appears to have developedproblems. For example, by invoking a help utility the receiver may becaused to automatically connected to the test computer, execute theself-test described above, and then transmit the results to amanufacturer computer for diagnostic use.

It will be appreciated that whilst present principals have beendescribed with reference to some example embodiments, these are notintended to be limiting, and that various alternative arrangements maybe used to implement the subject matter claimed herein.

1. A digital television system comprising: at least one receiverconfigured to receive digital television from at least one transmitterassembly, the receiver comprising: at least one processor programmedwith instructions to: execute at least a first broadcaster applicationto access test code in at least a first service; identify, based atleast in part on the test code, whether the receiver renders content;and transmit a signal to the transmitter assembly indicating whether thereceiver renders the content, wherein the test code comprises a commandto change service, wherein the command is to change from a first serviceto a second service, and the instructions are executable to use a firstapplication programming interface (API) in rendering the first serviceand a second API in rendering the second service.
 2. The digitaltelevision system of claim 1, comprising the transmitter assembly. 3.The digital television system of claim 1, wherein the digital televisionsystem comprises an advanced television systems committee (ATSC) 3.0system.
 4. The digital television system of claim 1, wherein the testcode comprises cascading style sheet (CSS) code. 5-6. (canceled)
 7. Adigital television system comprising: at least one receiver comprisingat least one processor programmed with instructions to configure theprocessor to: receive from broadcaster equipment a test application;execute the test application; load at least one entry page from at leastone web server onto at least one browser of the receiver; using the testapplication, execute API calls to exercise compliance with a standard;and use the test application to execute a first API command associatedwith a first broadcast channel and execute a second API commandassociated with a second broadcast channel to identify execution of acommand to change tuning from the first broadcast channel to the secondbroadcast channel.
 8. The digital television system of claim 7,comprising the broadcaster equipment.
 9. The digital television systemof claim 7, wherein the digital television system comprises an advancedtelevision systems committee (ATSC) 3.0 system.
 10. The digitaltelevision system of claim 7, wherein the instructions are executableto: implement an application programming interface (API) for loggingcommands and responses; execute the test application to exercise atleast one connection to at least one Web Socket Server and at least onehypertext transfer protocol (HTTP) interface.
 11. The digital televisionsystem of claim 10, wherein the instructions are executable to: executethe test application to test support of a user agent of the receiver forat least one web browser at least on part by sending a signal backthrough a hypertext transfer protocol (HTTP) interface to a testcomputer.
 12. The digital television system of claim 10, wherein theinstructions are executable to: execute the test application at least inpart by sending a ‘hello’ signal back to the broadcaster equipment. 13.The digital television system of claim 10, wherein the instructions areexecutable to: execute the test application at least in part bymaintaining in storage of the receiver a ‘hello’ signal.
 14. The digitaltelevision system of claim 7, wherein the instructions are executableto: execute the test application to identify proper execution of APIs ofat least one WebSocket of the receiver connected to at least onewebsocket server.
 15. (canceled)
 16. In a digital television system, amethod comprising: executing at least a first broadcaster application toaccess test code in at least a first service provided to a receiver;identifying, based at least in part on the test code, whether thereceiver renders content, wherein the first broadcaster application (BA)comprises a test configured with BA functionality but that is notprovided by a broadcaster but rather by a manufacturer of the receiverto run a test script in which the test BA self-generates commands tocomponents of the receiver and monitors for correct feedback.
 17. Themethod of claim 16, wherein the digital television system comprises anadvanced television systems committee (ATSC) 3.0 system.
 18. The methodof claim 16, comprising: responsive to a command to change from a firstservice to a second service, using a first application programminginterface (API) command in rendering the first service and a second APIcommand in rendering the second service.
 19. The method of claim 18,comprising: loading at least one entry page from at least one web serveronto at least one browser of the receiver.
 20. The method of claim 18,comprising: executing the first broadcaster application to exercise atleast one connection to at least one Web Socket Server, and at least onehypertext transfer protocol (HTTP) interface.